Hi,
Im merve, a total novice from istanbul. Guess ...
# chipseqm
MervevK
05/20/2024, 8:25 AMHi,
Im merve, a total novice from istanbul. Guess this is a very dumb question but i am really in doubt. I think structure of my csv for chipseq file is wrong. The thing is i have 48 fastq.gz files for 1 sample. So my the way i made csv is:
Sample name, 48 fastq files, antibody, control which is again ~40 fastq files. I get this error in the image. Can you help me please?
Wish you an awesome day
Best
MervevK
05/20/2024, 9:54 AMThis is what i got as execution trace: "task_id hash native_id name status exit submit duration realtime %cpu peak_rss peak_vmem rchar wchar
1 ca/a0309c 391295 NFCORE_CHIPSEQCHIPSEQINPUT_CHECK:SAMPLESHEET_CHECK (nfcore_trial.csv) FAILED 255 2024-05-20 103529.907 1.4s 1.3s - - - - -"
Guess my doubts are true.
How can i solve this issue?
a
Ashley Dederich
05/20/2024, 10:57 PMCould you provide a snippet of the csv file you provide?
m
MervevK
05/21/2024, 11:14 AM@Ashley Dederich
@Jose Espinosa-Carrasco sure, thank you for looking at it. Basically there are two problems:
1. csv /tsv does not read
2. there is a loop problem
1) for this i tried 3 different approaches:
a-Tried a very simple part of reads
b-Made either horizontal or vertical lists of fastq files (because there are +40 fastq.gz)
c-I concatenated all the fastq.gz files and tried like that
the scripts are like this:
exp.csv (1)
sample, fastq, antibody, control,
h3k4me1_ms_b_ENCSR084FXT /cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/ENCFF021LJN.fastq.gz /cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/ENCFF190MSV.fastq.gz /cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/ENCFF299MXX.fastq.gz /cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/ENCFF431PPA.fastq.gz /cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/ENCFF560XGR.fastq.gz /cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/ENCFF729ZSY.fastq.gz /cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/ENCFF852WSO.fastq.gz /cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/ENCFF033TOS.fastq.gz /cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/ENCFF192JGX.fastq.gz /cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/ENCFF320SCV.fastq.gz /cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/ENCFF450DWU.fastq.gz /cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/ENCFF587UTV.fastq.gz /cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/ENCFF740LGL.fastq.gz /cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/ENCFF865KWO.fastq.gz /cta/users/merve.kaftancioglu/ENCFF593SYY.fastq.gz
by the way they are separated by commas, it does not show here properly
exp.csv (2)
sample_ID,sample_read,Ab,control
h3k4me1_ms_b_ENCSR084FXT, "/cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR084FXT/h3k4me1_ms_b_ENCSR084FXT.fastq.gz", h3k4me1, "/cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_normal_b_ENCSR156BXM/h3k4me1_normal_b_ENCSR156BXM.fastq.gz"
"h3k4me1_ms_b_ENCSR480ITK, ""/cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_ms_b_ENCSR480ITK/h3k4me1_ms_b_ENCSR480ITK.fastq.gz"", h3k4me1, ""/cta/users/merve.kaftancioglu/alternative_scenario/seqs/h3k4me1_normal_b_ENCSR156BXM/h3k4me1_normal_b_ENCSR156BXM.fastq.gz"""
MervevK
05/21/2024, 11:23 AM@Ashley Dederich
@Jose Espinosa-Carrasco the output i get is:
Execution cancelled -- Finishing pending tasks before exit
-[nf-core/chipseq] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_CHIPSEQCHIPSEQINPUT_CHECK:SAMPLESHEET_CHECK (nfcore_trial_2.csv)'
Caused by:
Process
NFCORE_CHIPSEQ:CHIPSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (nfcore_trial_2.csv)cat .command.outNFCORE_CHIPSEQ:CHIPSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (nfcore_trial_2.csv)cat .command.outMervevK
05/21/2024, 11:25 AM@Ashley Dederich
@Jose Espinosa-Carrasco
2) I talked with IT guys from school, they suggested several things for the loop problem. they also mentioned some other people got this issues before, but they are good now after applying those suggestions. I tried everything, plus looked for the external support group of school's IT. at the end they said, they cannot solve it
MervevK
05/21/2024, 11:28 AM@Ashley Dederich
@Jose Espinosa-Carrasco I really appreciate all your supports. I am trying things all on my own, and this project I suppose to present 2 days later, had almost zero help from my prof or classmates. i am about the give up from everything. That really means a lot to me.
j
Jose Espinosa-Carrasco
05/21/2024, 11:45 AMThe header of the csv file is wrong, should be
antibody Copy code
ERROR: Please check samplesheet header -> sample_ID,sample_read,Ab,control != sample,fastq_1,fastq_2,antibody,controlm
MervevK
05/21/2024, 11:50 AMso my fastq concatenation approach was correct, I assume?
MervevK
05/21/2024, 12:19 PMcorrected everything again but same error happened:
"Execution cancelled -- Finishing pending tasks before exit
ERROR ~ Error executing process > 'NFCORE_CHIPSEQCHIPSEQINPUT_CHECK:SAMPLESHEET_CHECK (nfcore_trial_2.csv)'
Caused by:
Process
NFCORE_CHIPSEQ:CHIPSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (nfcore_trial_2.csv)bash .command.runNFCORE_CHIPSEQ:CHIPSEQ:INPUT_CHECK:SAMPLESHEET_CHECK (nfcore_trial_2.csv)j
Jose Espinosa-Carrasco
05/21/2024, 12:26 PMAgain your header is wrong, should be
fastq_2fast_2Jose Espinosa-Carrasco
05/21/2024, 12:27 PMAbout the samplesheet take a look to this
m
MervevK
05/21/2024, 1:36 PMthank you so much for answering my stupid mistakes, i am really overwhelmed after checking everything, i miss small stuff. This time i got a memory problem: "Execution cancelled -- Finishing pending tasks before exit
-[nf-core/chipseq] Pipeline completed with errors-
ERROR ~ Error executing process > 'NFCORE_CHIPSEQCHIPSEQFASTQC_TRIMGALORE:FASTQC (h3k4me1_ms_b_ENCSR084FXT_T1)'
Caused by:
Process requirement exceeds available memory -- req: 36 GB; avail: 31.2 GB
Command executed:
[ ! -f h3k4me1_ms_b_ENCSR084FXT_T1.fastq.gz ] && ln -s h3k4me1_ms_b_ENCSR084FXT.fastq.gz h3k4me1_ms_b_ENCSR084FXT_T1.fastq.gz
fastqc --quiet --threads 6 h3k4me1_ms_b_ENCSR084FXT_T1.fastq.gz
cat <<-END_VERSIONS > versions.yml
"NFCORE_CHIPSEQCHIPSEQFASTQC_TRIMGALORE:FASTQC":
fastqc: $( fastqc --version | sed -e "s/FastQC v//g" )
END_VERSIONS
Command exit status:
-
Command output:
(empty)
Work dir:
/cta/users/merve.kaftancioglu/alternative_scenario/nf-core/work/b0/80fa660909c70448e253b8dbba11c4
Tip: view the complete command output by changing to the process work dir and entering the command
cat .command.outj
Jose Espinosa-Carrasco
05/21/2024, 1:57 PMNot sure how is the infrastructure where you are running the pipeline, you could restrict to the available memory in your system by adding to your nextflow command this parameter
--max_memory 30.GB👍 1
Jose Espinosa-Carrasco
05/21/2024, 1:58 PMNot sure if still you will need more memory to run the pipeline with your samples
👍 1
a
Alexandru Mizeranschi
05/21/2024, 2:02 PMHey @Jose Espinosa-Carrasco
Regarding the option
--max_memoryprocess_mediumprocess_medium36.GB--max_memory 30.GB30.GB👍🏻 1
👍 1
Alexandru Mizeranschi
05/21/2024, 2:16 PM@MervevK to get the
FASTQCcustom.config Copy code
process {
withName: 'FASTQC' {
memory = 30.GB
}
}-c custom.configFASTQC👍 1
m
MervevK
05/21/2024, 2:19 PMDo i need to add anything else?
a
Alexandru Mizeranschi
05/21/2024, 2:50 PMHow many CPU cores does your machine have?
m
MervevK
05/21/2024, 2:53 PMApparently 8, now i got cpu error
a
Alexandru Mizeranschi
05/21/2024, 2:57 PMplease copy and paste here the whole list of processes that was configured by the pipeline. it should look something like this (but with different process names for you):
Copy code
[45/0cc0e2] process > NFCORE_FUNCSCAN:FUNCSCAN:GUNZIP_FASTA_PREP (MEGAHIT-SRR16971104.contigs.fa.gz) [100%] 2 of 2 ✔
[2e/a303e3] process > NFCORE_FUNCSCAN:FUNCSCAN:SEQKIT_SEQ_LONG (SRR16971104) [100%] 2 of 2 ✔
[df/feffe9] process > NFCORE_FUNCSCAN:FUNCSCAN:SEQKIT_SEQ_SHORT (SRR16971104) [100%] 2 of 2 ✔
[a7/c71888] process > NFCORE_FUNCSCAN:FUNCSCAN:PYRODIGAL_GFF (SRR16971104_short) [100%] 4 of 4 ✔
[1a/6e7e15] process > NFCORE_FUNCSCAN:FUNCSCAN:GUNZIP_PYRODIGAL_FAA (SRR16971104_short.faa.gz) [100%] 4 of 4 ✔
[d9/35cfb2] process > NFCORE_FUNCSCAN:FUNCSCAN:GUNZIP_PYRODIGAL_FNA (SRR16971104_short.fna.gz) [100%] 4 of 4 ✔
[2d/26a9e2] process > NFCORE_FUNCSCAN:FUNCSCAN:GUNZIP_PYRODIGAL_GFF (SRR16971104_short.gff.gz) [100%] 4 of 4 ✔
[39/236abf] process > NFCORE_FUNCSCAN:FUNCSCAN:ARG:UNTAR_CARD (data) [100%] 1 of 1 ✔
[e1/101700] process > NFCORE_FUNCSCAN:FUNCSCAN:ARG:RGI_CARDANNOTATION [100%] 1 of 1 ✔
[75/5be86c] process > NFCORE_FUNCSCAN:FUNCSCAN:ARG:RGI_MAIN (SRR16971104_short) [100%] 4 of 4 ✔
[56/b071cd] process > NFCORE_FUNCSCAN:FUNCSCAN:ARG:HAMRONIZATION_RGI (SRR16971104_short) [100%] 4 of 4 ✔
[84/ebf3d5] process > NFCORE_FUNCSCAN:FUNCSCAN:ARG:DEEPARG_DOWNLOADDATA [100%] 1 of 1 ✔
[98/7d5d78] process > NFCORE_FUNCSCAN:FUNCSCAN:ARG:DEEPARG_PREDICT (SRR16971104_short) [100%] 4 of 4 ✔
[3a/0ec18d] process > NFCORE_FUNCSCAN:FUNCSCAN:ARG:HAMRONIZATION_DEEPARG (SRR16971104_short) [100%] 8 of 8 ✔
[fc/41a051] process > NFCORE_FUNCSCAN:FUNCSCAN:ARG:HAMRONIZATION_SUMMARIZE [100%] 1 of 1 ✔
[97/f6995e] process > NFCORE_FUNCSCAN:FUNCSCAN:MULTIQC [100%] 1 of 1 ✔m
MervevK
05/21/2024, 3:02 PMI really appreciate this, thank you so much
a
Alexandru Mizeranschi
05/21/2024, 3:04 PMJust to make sure, I meant if you could please copy and paste here on Slack the list of process that you see on your machine
Alexandru Mizeranschi
05/21/2024, 3:05 PMI showed you an example above. you can see a few of the processes in the top part of the screenshot that you posted above
m
MervevK
05/21/2024, 3:08 PMI will try
MervevK
05/21/2024, 3:08 PMThank you so much
a
Alexandru Mizeranschi
05/21/2024, 3:20 PMTry the following contents below in your
custom.config Copy code
process {
withLabel: process_single {
cpus = 1
memory = 4.GB
time = 24h
}
withLabel: process_low {
cpus = 2
memory = 7.GB
time = 24h
}
withLabel: process_medium {
cpus = 8
memory = 30.GB
time = 24h
}
withLabel: process_high {
cpus = 8
memory = 30.GB
time = 24h
}
withLabel: process_long {
cpus = 8
memory = 30.GB
time = 24h
}
}m
MervevK
05/22/2024, 3:48 PMyesterday i couldn't have a chance to try because i couldn't reach to cluster, but i will now, thank you so much
MervevK
05/22/2024, 4:29 PMafter i created the config file i got a very weird error:
"ERROR ~ ERROR: Validation of pipeline parameters failed!
-- Check '.nextflow.log' file for details
ERROR ~ * --input: string [/cta/users/merve.kaftancioglu/seqs/h3k4me1_ms_b_ENCSR084FXT/h3k4me1_ms_b_ENCSR084FXT.fastq.gz] does not match pattern ^\S+\.csv$ (/cta/users/merve.kaftancioglu/seqs/h3k4me1_ms_b_ENCSR084FXT/h3k4me1_ms_b_ENCSR084FXT.fastq.gz)
-- Check '.nextflow.log' file for details"
it was at the beginning of the analysis, did not even load anything. what could be the cause, do you have any idea?
a
Ashley Dederich
05/22/2024, 4:36 PMCan you copy-paste your 'nextflow run nf-core/chipseq' command? Looks like you supplied the wrong input file.
m
MervevK
05/22/2024, 4:42 PMthank you @Ashley Dederich. this is the command:
##nf-core
nextflow run nf-core/chipseq --input /cta/users/merve.kaftancioglu/alternative_scenario/nf-core/nfcore_trial_2.csv --outdir /cta/users/merve.kaftancioglu/alternative_scenario/nf-core/output --genome GRCh38 -profile singularity --read_length 75 -c /cta/users/merve.kaftancioglu/alternative_scenario/nf-core/custom.config
MervevK
05/22/2024, 4:43 PMuntil now, i never had such an error
MervevK
05/22/2024, 4:52 PMi tried once more, this time nextflow makes a glitch: "N E X T F L O W ~ version 23.10.1
ERROR ~ Unknown method invocation
calla
Ashley Dederich
05/22/2024, 4:59 PMAre those quotation marks (") in the nextflow.config file? Also, you do not need to put single quotes around the names after withName or withLabel
m
MervevK
05/22/2024, 5:00 PMi have added the quotes
a
Ashley Dederich
05/22/2024, 5:00 PMThe quotes do not need to be in the nextflow.config file.
m
MervevK
05/22/2024, 5:01 PMi didn't put in the original file, for FASTQC single quotes, it was the way Alexandru has shared, now i am removing them too
MervevK
05/22/2024, 5:06 PMit happened again:
"N E X T F L O W ~ version 23.10.1
ERROR ~ Unknown method invocation
calla
Ashley Dederich
05/22/2024, 8:59 PMWhat does the end of the .nextflow.log file look like?
m
MervevK
05/23/2024, 10:26 AMDear @Ashley Dederich @Jose Espinosa-Carrasco @Phil Ewels @Alexandru Mizeranschi I am really so thankful for your helps and guidance. Yesterday I played around AI to correct the config file. In my last attempt the script was this: "process {
withLabel: fastqc {
cpus = 1
memory = "30.GB"
time = "24h"
}
withLabel: process_low {
cpus = 2
memory = "7.GB"
time = "24h"
}
withLabel: process_medium {
cpus = 4
memory = "16.GB"
time = "24h"
}
withLabel: process_high {
cpus = 8
memory = "30.GB"
time = "24h"
}
withLabel: process_long {
cpus = 4
memory = "16.GB"
time = "48h"
}
}"
and it worked🤩
I want to make a contribution in monetary terms to nf-core, do you have a platform that I can do this? for technical contributions, I am still too inadequate, but I would love to support other people either.
Again, thank you so much everyone🌺
❤️ 2
p
Phil Ewels
05/23/2024, 11:05 AMLovely of you to think of it @MervevK - not at the moment though sorry (maybe in the future). For now the best gift is to share the love on social media / spread the word 😊
❤️ 1
m
MervevK
05/23/2024, 11:24 AMI definitely will.
Hope to converse on nf-core in future,
Till then
Best⚘️
🙏🏻 2